Project Outline¶
Data Loading and Preprocessing
- Load the dataset.
- Exploratory Data Analysis:
- Feature histograms.
- Train-Test Split (to avoid leakage).
- Check for outliers (and optional removal).
- Scaling (Standardization).
- Correlation Matrix and Multicollinearity Check:
- Remove highly correlated features.
- Prepare final features for analysis.
Linear Algorithms
- Perceptron
- Train with 5-fold cross-validation (CV).
- Hyperparameter tuning using CV.
- Evaluate the best-tuned model on the test set.
- Pegasos SVM
- Train with 5-fold cross-validation.
- Evaluate the model on the test set.
- Logistic Regression
- Train with 5-fold cross-validation.
- Evaluate the model on the test set.
- Comparison of Results
- Summarize and compare accuracies across Perceptron, Pegasos SVM, and Logistic Regression.
- Perceptron
Polynomial Feature Expansion
- Expand features to degree 2 (include squared terms and interactions).
- Retrain and evaluate:
- Perceptron with polynomial expansion.
- Pegasos SVM with polynomial expansion.
- Logistic Regression with polynomial expansion.
- Summarize and compare results:
- Linear vs. Polynomial-expanded features.
- Visualize results.
Kernelized Methods
- Kernelized Perceptron
- Implement and evaluate with:
- Gaussian Kernel.
- Polynomial Kernel.
- Compare test set accuracies.
- Implement and evaluate with:
- Kernelized Pegasos SVM
- Implement and evaluate with:
- Gaussian Kernel.
- Polynomial Kernel.
- Compare test set accuracies.
- Implement and evaluate with:
- Kernelized Perceptron
Final Comparison
- Summarize final results:
- Linear models.
- Polynomial-expanded models.
- Kernelized models.
- Summarize final results:
Kernelized Linear Classification¶
Initial imports + functions¶
Basic libraries + custom functions (ffunctions.py):
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# Uncomment the following line if you want that plots appear inline in a real Jupyter notebook:
# %matplotlib inline
from ffunctions import (
load_data,
train_test_split,
detect_outliers_zscore,
remove_outliers,
standard_scaler_fit,
standard_scaler_transform,
check_high_correlation,
k_fold_cross_validation,
perceptron_classifier_func,
pegasos_classifier_func,
logistic_regression_classifier_func,
perceptron_classifier_func_eta,
perceptron_train,
perceptron_predict,
pegasos_train,
pegasos_predict,
logistic_regression_train,
logistic_regression_predict,
polynomial_feature_expansion,
gaussian_kernel,
polynomial_kernel,
kernelized_perceptron_classifier_func,
kernelized_perceptron_train,
kernelized_perceptron_predict,
kernelized_pegasos_classifier_func,
kernelized_pegasos_train,
kernelized_pegasos_predict
)
best_hparams_table = {
"Perceptron": {},
"Pegasos": {},
"Logistic": {},
"PerceptronPoly": {},
"PegasosPoly": {},
"LogisticPoly": {},
"KernelPerceptronGauss": {},
"KernelPerceptronPoly": {},
"KernelPegasosGauss": {},
"KernelPegasosPoly": {}
}
Data loading:
csv_path = "/Users/michelecoaro/Documents/GitHub/KLC/klc_project/dataset.csv"
df = load_data(csv_path)
df.head()
| x1 | x2 | x3 | x4 | x5 | x6 | x7 | x8 | x9 | x10 | y | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1.205492 | 5.823226 | 98.837539 | -1.075852 | 0.999205 | 0.911543 | 3.623558 | -1.720267 | -0.346191 | -54.708330 | -1 |
| 1 | 1.692571 | -0.887019 | 100.901276 | -0.595548 | 0.177550 | -0.915495 | 4.320264 | 0.907834 | 3.126815 | -56.397484 | -1 |
| 2 | 4.289320 | 1.416843 | 100.784735 | -2.897154 | -0.066972 | -0.786173 | 2.093003 | 1.336237 | 2.183829 | -56.197728 | 1 |
| 3 | 0.542420 | -1.010095 | 100.015580 | -3.070705 | 0.088324 | -0.242669 | 0.767942 | -0.284683 | -2.104145 | -55.794045 | 1 |
| 4 | 2.431765 | -1.224177 | 100.709237 | -0.102892 | 0.699031 | -0.656315 | -0.911784 | -1.355873 | 1.537825 | -55.917863 | -1 |
Exploratory Data Analysis¶
Understanding feature locations and shape
feature_cols = [f"x{i}" for i in range(1, 11)]
df[feature_cols].hist(bins=30, figsize=(15, 10))
plt.suptitle("Histograms of Numerical Features", fontsize=16)
plt.show()
#pairplot of features with different colors for each feature in the pair
import seaborn as sns
sns.pairplot(df, hue='y', diag_kind='hist')
plt.suptitle("Pairplot of Numerical Features", fontsize=16)
plt.show()
Train - Test split as first data leakage countermeasure:
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
print("Train set shape:", train_df.shape)
print("Test set shape:", test_df.shape)
Train set shape: (7788, 11) Test set shape: (1947, 11)
Checking for outliers in training set
outlier_indices = detect_outliers_zscore(train_df, feature_cols, z_thresh=3.0)
print(f"Number of outliers detected in training set: {len(outlier_indices)}")
Number of outliers detected in training set: 82
Outliers removal:
train_df_no_outliers = remove_outliers(train_df, outlier_indices)
print("Train set (no outliers) shape:", train_df_no_outliers.shape)
Train set (no outliers) shape: (7706, 11)
Applying consistent scaling
means, stds = standard_scaler_fit(train_df_no_outliers, feature_cols)
train_scaled = standard_scaler_transform(train_df_no_outliers, feature_cols, means, stds)
test_scaled = standard_scaler_transform(test_df, feature_cols, means, stds)
# histogram plot for numerical variables after scaling
train_scaled[feature_cols].hist(bins=30, figsize=(15, 10))
plt.suptitle("Histograms of Numerical Features (After Scaling)", fontsize=16)
plt.show()
#side by side plot of features before and after scaling
fig, axs = plt.subplots(2, 5, figsize=(20, 8))
fig.suptitle("Side-by-Side Histograms of Numerical Features Before and After Scaling", fontsize=16)
for i, col in enumerate(feature_cols):
train_df[col].hist(bins=30, ax=axs[i // 5, i % 5])
axs[i // 5, i % 5].set_title(f"Before Scaling: {col}")
fig, axs = plt.subplots(2, 5, figsize=(20, 8))
for i, col in enumerate(feature_cols):
train_scaled[col].hist(bins=30, ax=axs[i // 5, i % 5])
axs[i // 5, i % 5].set_title(f"After Scaling: {col}")
plt.show()
Correlation Matrix
# 5. Correlation Matrix (for numerical columns)
corr_matrix = train_scaled[feature_cols].corr()
plt.figure(figsize=(8, 6))
plt.imshow(corr_matrix, cmap='coolwarm', aspect='auto')
plt.colorbar(label='Correlation Coefficient')
plt.title('Correlation Matrix')
plt.xticks(range(len(corr_matrix.columns)), corr_matrix.columns, rotation=45)
plt.yticks(range(len(corr_matrix.columns)), corr_matrix.columns)
plt.tight_layout()
plt.show()
Multicollinearity Check
high_corr_pairs = check_high_correlation(train_df_no_outliers, feature_cols, corr_threshold=0.9)
if high_corr_pairs:
print("High correlation pairs detected (above threshold):")
for pair in high_corr_pairs:
print(f" {pair[0]} and {pair[1]} with correlation = {pair[2]:.3f}")
else:
print("No pairs of features exceed the correlation threshold.")
High correlation pairs detected (above threshold): x3 and x6 with correlation = 0.990 x3 and x10 with correlation = 0.980 x6 and x10 with correlation = 0.990
Removing unwanted columns
for col in ["x3", "x6", "x10"]:
corr = train_df_no_outliers[col].corr(train_df_no_outliers["y"])
print(col, corr)
final_features = [f for f in feature_cols if f not in ["x3", "x6"]]
print("\nDropping x3, x6 from the final feature set...")
print("Final features:", final_features)
x3 0.051673811986996 x6 -0.05343499561215009 x10 -0.05288351228543469 Dropping x3, x6 from the final feature set... Final features: ['x1', 'x2', 'x4', 'x5', 'x7', 'x8', 'x9', 'x10']
Algorithms Implementation¶
Cross-validation¶
target_col = "y"
# Example: Perceptron with 5 epochs
accuracy_perc = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=5,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: perceptron_classifier_func(
tr_fold, vl_fold, f, t, epochs=5
)
)
print(f"Perceptron 5-fold CV Accuracy: {accuracy_perc:.3f}")
# Example: Pegasos
accuracy_pegasos = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=5,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: pegasos_classifier_func(
tr_fold, vl_fold, f, t, lambda_param=0.01, epochs=5
)
)
print(f"Pegasos 5-fold CV Accuracy: {accuracy_pegasos:.3f}")
# Example: Logistic Regression
accuracy_logreg = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=5,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: logistic_regression_classifier_func(
tr_fold, vl_fold, f, t, lambda_param=0.01, epochs=5, eta=1.0
)
)
print(f"Logistic Regression 5-fold CV Accuracy: {accuracy_logreg:.3f}")
Perceptron 5-fold CV Accuracy: 0.585 Pegasos 5-fold CV Accuracy: 0.723 Logistic Regression 5-fold CV Accuracy: 0.718
Perceptron¶
Hyperparameter Tuning¶
target_col = "y"
epochs_list = [5, 10, 20, 50]
learning_rates = [0.01, 0.1, 1.0]
tuning_results_perc = {}
for ep in epochs_list:
for eta in learning_rates:
acc = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=5,
random_state=42,
classifier_func=lambda tr, vl, f, t: perceptron_classifier_func_eta(
tr, vl, f, t, epochs=ep, eta=eta
)
)
tuning_results_perc[(ep, eta)] = acc
best_params_perc = max(tuning_results_perc, key=tuning_results_perc.get)
best_epochs_perc, best_eta_perc = best_params_perc
best_acc_perc = tuning_results_perc[best_params_perc]
print("Perceptron best hyperparams =>", best_params_perc, "CV Accuracy =", best_acc_perc)
# Save in best_hparams_table
best_hparams_table["Perceptron"]["epochs"] = best_epochs_perc
best_hparams_table["Perceptron"]["eta"] = best_eta_perc
best_hparams_table["Perceptron"]["cv_accuracy"] = best_acc_perc
# Train final model on entire train set with best hyperparams
from ffunctions import perceptron_train_eta, perceptron_predict
theta_perc, theta0_perc = perceptron_train_eta(
train_scaled, final_features, target_col,
epochs=best_epochs_perc, eta=best_eta_perc
)
test_preds_perc = perceptron_predict(test_scaled, final_features, theta_perc, theta0_perc)
test_acc_perc = np.mean(test_preds_perc == test_scaled[target_col])
print(f"Perceptron final test accuracy (best hyperparams): {test_acc_perc:.3f}")
Perceptron best hyperparams => (10, 0.1) CV Accuracy = 0.6466382349797284 Perceptron final test accuracy (best hyperparams): 0.601
# Compare default and tuned Perceptron across feature subsets
subsets = {
"Features": feature_cols}
comparison_results = {}
for subset_name, subset_features in subsets.items():
print(f"\nEvaluating subset: {subset_name}")
# Default Perceptron
accuracy_default_perc = k_fold_cross_validation(
train_scaled,
features=subset_features,
target='y',
k=5,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: perceptron_classifier_func(
tr_fold, vl_fold, f, t, epochs=5 # Default epochs
)
)
# Tuned Perceptron
accuracy_tuned_perc = k_fold_cross_validation(
train_scaled,
features=subset_features,
target='y',
k=5,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: perceptron_classifier_func_eta(
tr_fold, vl_fold, f, t, epochs=best_epochs_perc, eta=best_eta_perc
)
)
comparison_results[subset_name] = {
"Default Perceptron": accuracy_default_perc,
"Tuned Perceptron": accuracy_tuned_perc,
}
print(f" Default Perceptron Accuracy: {accuracy_default_perc:.3f}")
print(f" Tuned Perceptron Accuracy: {accuracy_tuned_perc:.3f}")
Evaluating subset: Features Default Perceptron Accuracy: 0.624 Tuned Perceptron Accuracy: 0.627
Prediction on test dataset
theta_perc, theta0_perc = perceptron_train(
train_scaled,
features=final_features,
target=target_col,
epochs=5
)
test_preds_perc = perceptron_predict(
test_scaled,
features=final_features,
theta=theta_perc,
theta_0=theta0_perc
)
test_acc_perc = np.mean(test_preds_perc == test_scaled[target_col])
print(f"Linear Perceptron final test accuracy: {test_acc_perc:.3f}")
Linear Perceptron final test accuracy: 0.670
Pegasos SVM¶
Hyperparameter Tuning
lambda_values_pega = [0.001, 0.01, 0.1]
epochs_values_pega = [5, 10, 20]
tuning_results_pegasos = {}
for lam in lambda_values_pega:
for e in epochs_values_pega:
acc = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=5,
random_state=42,
classifier_func=lambda tr, vl, f, t: pegasos_classifier_func(
tr, vl, f, t, lambda_param=lam, epochs=e
)
)
tuning_results_pegasos[(lam, e)] = acc
best_lambda_pega, best_epochs_pega = max(tuning_results_pegasos, key=tuning_results_pegasos.get)
best_acc_pega_cv = tuning_results_pegasos[(best_lambda_pega, best_epochs_pega)]
print("Pegasos best hyperparams =>", (best_lambda_pega, best_epochs_pega), "CV =", best_acc_pega_cv)
best_hparams_table["Pegasos"]["lambda"] = best_lambda_pega
best_hparams_table["Pegasos"]["epochs"] = best_epochs_pega
best_hparams_table["Pegasos"]["cv_accuracy"] = best_acc_pega_cv
# Train final Pegasos
theta_pega, theta0_pega = pegasos_train(
train_scaled, final_features, target_col,
lambda_param=best_lambda_pega, epochs=best_epochs_pega
)
test_preds_pega = pegasos_predict(test_scaled, final_features, theta_pega, theta0_pega)
test_acc_pega = np.mean(test_preds_pega == test_scaled[target_col])
print(f"Pegasos final test accuracy (best hyperparams): {test_acc_pega:.3f}")
Pegasos best hyperparams => (0.001, 20) CV = 0.7245005727579326 Pegasos final test accuracy (best hyperparams): 0.741
Comparison of different subsets¶
# Use the best hyperparameters found during tuning
lambda_param_pega = best_lambda_pega
epochs_pega = best_epochs_pega
# Compare Pegasos SVM across feature subsets
pegasos_results = {}
for subset_name, subset_features in subsets.items():
print(f"\nEvaluating Pegasos SVM on subset: {subset_name}")
accuracy_pegasos = k_fold_cross_validation(
train_scaled,
features=subset_features,
target='y',
k=5,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: pegasos_classifier_func(
tr_fold, vl_fold, f, t, lambda_param=lambda_param_pega, epochs=epochs_pega
)
)
pegasos_results[subset_name] = accuracy_pegasos
print(f" Pegasos SVM Accuracy: {accuracy_pegasos:.3f}")
Evaluating Pegasos SVM on subset: Features Pegasos SVM Accuracy: 0.711
Measring accuracy of prediction¶
By testing on test set
theta_pega, theta0_pega = pegasos_train(
train_scaled,
features=final_features,
target=target_col,
epochs=5
)
test_preds_pega = pegasos_predict(
test_scaled,
features=final_features,
theta=theta_pega,
theta_0=theta0_pega
)
test_acc_pega = np.mean(test_preds_pega == test_scaled[target_col])
print(f"Pegasos final test accuracy: {test_acc_pega:.3f}")
Pegasos final test accuracy: 0.733
Logistic Regression¶
Hyperparameter tuning
lambda_list_logreg = [0.001, 0.01, 0.1]
epochs_list_logreg = [5, 10]
eta_list_logreg = [0.01, 0.1, 1.0]
tuning_results_logreg = {}
for lam in lambda_list_logreg:
for ep in epochs_list_logreg:
for eta in eta_list_logreg:
acc = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=5,
random_state=42,
classifier_func=lambda tr, vl, f, t: logistic_regression_classifier_func(
tr, vl, f, t,
lambda_param=lam,
epochs=ep,
eta=eta
)
)
tuning_results_logreg[(lam, ep, eta)] = acc
best_lam_logreg, best_ep_logreg, best_eta_logreg = max(tuning_results_logreg, key=tuning_results_logreg.get)
best_acc_logreg_cv = tuning_results_logreg[(best_lam_logreg, best_ep_logreg, best_eta_logreg)]
print("Logistic best hyperparams =>", (best_lam_logreg, best_ep_logreg, best_eta_logreg), "CV =", best_acc_logreg_cv)
best_hparams_table["Logistic"]["lambda"] = best_lam_logreg
best_hparams_table["Logistic"]["epochs"] = best_ep_logreg
best_hparams_table["Logistic"]["eta"] = best_eta_logreg
best_hparams_table["Logistic"]["cv_accuracy"] = best_acc_logreg_cv
# Train final logistic
theta_lr, theta0_lr = logistic_regression_train(
train_scaled, final_features, target_col,
lambda_param=best_lam_logreg,
epochs=best_ep_logreg,
eta=best_eta_logreg
)
test_preds_lr = logistic_regression_predict(test_scaled, final_features, theta_lr, theta0_lr)
test_acc_lr = np.mean(test_preds_lr == test_scaled[target_col])
print(f"Logistic final test accuracy (best hyperparams): {test_acc_lr:.3f}")
Logistic best hyperparams => (0.1, 10, 1.0) CV = 0.7173635291652042 Logistic final test accuracy (best hyperparams): 0.729
# Define parameters for Logistic Regression
lambda_param_logreg = best_lam_logreg
epochs_logreg = best_ep_logreg
eta_logreg = best_eta_logreg
# Compare Logistic Regression across feature subsets
logistic_results = {}
for subset_name, subset_features in subsets.items():
print(f"\nEvaluating Logistic Regression on subset: {subset_name}")
accuracy_logistic = k_fold_cross_validation(
train_scaled,
features=subset_features,
target='y',
k=5,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: logistic_regression_classifier_func(
tr_fold, vl_fold, f, t, lambda_param=best_lam_logreg, epochs=best_ep_logreg, eta=best_eta_logreg
)
)
logistic_results[subset_name] = accuracy_logistic
print(f" Logistic Regression Accuracy: {accuracy_logistic:.3f}")
Evaluating Logistic Regression on subset: Features Logistic Regression Accuracy: 0.714
# Train the Logistic Regression model
theta_lr, theta0_lr = logistic_regression_train(
train_scaled,
features=final_features,
target=target_col,
lambda_param=best_lam_logreg, # Regularization parameter
epochs=best_ep_logreg,
eta=best_eta_logreg # Learning rate
)
# Predict on the test set
test_preds_lr = logistic_regression_predict(
test_scaled,
features=final_features,
theta=theta_lr,
theta_0=theta0_lr
)
# Compute accuracy
test_acc_lr = np.mean(test_preds_lr == test_scaled[target_col])
print(f"Logistic Regression final test accuracy: {test_acc_lr:.3f}")
Logistic Regression final test accuracy: 0.734
Comparison¶
# Combine results into a DataFrame
comparison_with_logistic = pd.DataFrame({
"Default Perceptron Loss": [comparison_results[name]["Default Perceptron"] for name in subsets.keys()],
"Tuned Perceptron Accuracy": [comparison_results[name]["Tuned Perceptron"] for name in subsets.keys()],
"Pegasos SVM Accuracy": [pegasos_results[name] for name in subsets.keys()],
"Logistic Regression Accuracy": [logistic_results[name] for name in subsets.keys()]
}, index=list(subsets.keys()))
# Display the comparison table
print("\nComparison of Perceptron, Pegasos, and Logistic Regression on training set:")
display(comparison_with_logistic)
# Comparison table on test set
comparison_test = pd.DataFrame({
"Perceptron (Default)": [test_acc_perc],
"Perceptron (Tuned)": [comparison_results["Features"]["Tuned Perceptron"]],
"Pegasos SVM": [test_acc_pega],
"Logistic Regression": [test_acc_lr]
}, index=["Test Set"])
# Display the comparison table
print("\nComparison of Perceptron, Pegasos, and Logistic Regression on test set:")
display(comparison_test)
Comparison of Perceptron, Pegasos, and Logistic Regression on training set:
| Default Perceptron Loss | Tuned Perceptron Accuracy | Pegasos SVM Accuracy | Logistic Regression Accuracy | |
|---|---|---|---|---|
| Features | 0.62392 | 0.626535 | 0.711005 | 0.714379 |
Comparison of Perceptron, Pegasos, and Logistic Regression on test set:
| Perceptron (Default) | Perceptron (Tuned) | Pegasos SVM | Logistic Regression | |
|---|---|---|---|---|
| Test Set | 0.670262 | 0.626535 | 0.732922 | 0.73395 |
Polynomial feature expansion¶
# Expand the scaled training data (train_scaled) and test data (test_scaled)
# using the final_features (excluding x3, x6). Then rescale the newly created polynomial columns.
# 1) Expand polynomial features on the *scaled* train and test data.
train_poly_unscaled = polynomial_feature_expansion(train_scaled, final_features, degree=2, include_bias=False)
test_poly_unscaled = polynomial_feature_expansion(test_scaled, final_features, degree=2, include_bias=False)
# Identify new polynomial columns (excluding 'y')
expanded_feature_cols = [col for col in train_poly_unscaled.columns if col != 'y']
print("Polynomial expansion complete.")
print("Number of expanded features:", len(expanded_feature_cols))
print("Sample of expanded features:", expanded_feature_cols[:10])
Polynomial expansion complete. Number of expanded features: 46 Sample of expanded features: ['x1', 'x1^2', 'x2', 'x2^2', 'x4', 'x4^2', 'x5', 'x5^2', 'x7', 'x7^2']
# 2) Rescale the newly expanded features
means_poly, stds_poly = standard_scaler_fit(train_poly_unscaled, expanded_feature_cols)
train_poly = standard_scaler_transform(train_poly_unscaled, expanded_feature_cols, means_poly, stds_poly)
test_poly = standard_scaler_transform(test_poly_unscaled, expanded_feature_cols, means_poly, stds_poly)
k = 5
target_col = "y"
Perceptron algorithm with polynomial expansion¶
acc_perc_poly = k_fold_cross_validation(
df=train_poly,
features=expanded_feature_cols,
target=target_col,
k=5,
random_state=42,
classifier_func=lambda tr, vl, f, t: perceptron_classifier_func_eta(
tr, vl, f, t, epochs=best_epochs_perc, eta=best_eta_perc
)
)
theta_perc_poly, theta0_perc_poly = perceptron_train_eta(
train_poly, expanded_feature_cols, target_col,
epochs=best_epochs_perc, eta=best_eta_perc
)
test_preds_poly_perc = perceptron_predict(
test_poly, expanded_feature_cols, theta_perc_poly, theta0_perc_poly
)
test_acc_poly_perc = np.mean(test_preds_poly_perc == test_poly[target_col])
print("\nPerceptron + Poly => CV=%.3f, Test=%.3f" % (acc_perc_poly, test_acc_poly_perc))
best_hparams_table["PerceptronPoly"]["epochs"] = best_epochs_perc
best_hparams_table["PerceptronPoly"]["eta"] = best_eta_perc
best_hparams_table["PerceptronPoly"]["cv_accuracy"] = acc_perc_poly
Perceptron + Poly => CV=0.926, Test=0.924
SVM with PE¶
lambda_list_poly = [0.001, 0.01, 0.1]
epochs_list_poly = [5, 10, 20]
tuning_results_pegasos_poly = {}
for lam in lambda_list_poly:
for ep in epochs_list_poly:
acc = k_fold_cross_validation(
df=train_poly,
features=expanded_feature_cols,
target=target_col,
k=5,
random_state=42,
classifier_func=lambda tr, vl, f, t: pegasos_classifier_func(
tr, vl, f, t, lambda_param=lam, epochs=ep
)
)
tuning_results_pegasos_poly[(lam, ep)] = acc
best_lam_pega_poly, best_ep_pega_poly = max(tuning_results_pegasos_poly, key=tuning_results_pegasos_poly.get)
acc_pega_poly = tuning_results_pegasos_poly[(best_lam_pega_poly, best_ep_pega_poly)]
print("Pegasos + Poly best hyperparams =>", (best_lam_pega_poly, best_ep_pega_poly), "CV =", acc_pega_poly)
best_hparams_table["PegasosPoly"]["lambda"] = best_lam_pega_poly
best_hparams_table["PegasosPoly"]["epochs"] = best_ep_pega_poly
best_hparams_table["PegasosPoly"]["cv_accuracy"] = acc_pega_poly
# Train final Pegasos on polynomial
theta_pega_poly, theta0_pega_poly = pegasos_train(
train_poly, expanded_feature_cols, target_col,
lambda_param=best_lam_pega_poly, epochs=best_ep_pega_poly
)
test_preds_pega_poly = pegasos_predict(
test_poly, expanded_feature_cols, theta_pega_poly, theta0_pega_poly
)
test_acc_pega_poly = np.mean(test_preds_pega_poly == test_poly[target_col])
print(f"Pegasos + Poly final test accuracy: {test_acc_pega_poly:.3f}")
Pegasos + Poly best hyperparams => (0.001, 20) CV = 0.9457567516839758 Pegasos + Poly final test accuracy: 0.949
Logistic Regression with PE¶
Training set cross validation:
acc_logreg_poly = k_fold_cross_validation(
df=train_poly,
features=expanded_feature_cols,
target=target_col,
k=5,
random_state=42,
classifier_func=lambda tr, vl, f, t: logistic_regression_classifier_func(
tr, vl, f, t,
lambda_param=best_lam_logreg,
epochs=best_ep_logreg,
eta=best_eta_logreg
)
)
theta_logreg_poly, theta0_logreg_poly = logistic_regression_train(
train_poly, expanded_feature_cols, target_col,
lambda_param=best_lam_logreg,
epochs=best_ep_logreg,
eta=best_eta_logreg
)
test_preds_logreg_poly = logistic_regression_predict(
test_poly, expanded_feature_cols, theta_logreg_poly, theta0_logreg_poly
)
test_acc_logreg_poly = np.mean(test_preds_logreg_poly == test_poly[target_col])
print("LogReg + Poly => CV=%.3f, Test=%.3f" % (acc_logreg_poly, test_acc_logreg_poly))
best_hparams_table["LogisticPoly"]["lambda"] = best_lam_logreg
best_hparams_table["LogisticPoly"]["epochs"] = best_ep_logreg
best_hparams_table["LogisticPoly"]["eta"] = best_eta_logreg
best_hparams_table["LogisticPoly"]["cv_accuracy"] = acc_logreg_poly
LogReg + Poly => CV=0.882, Test=0.882
Results¶
print("\n=== Polynomial-Expanded Feature Set Accuracies ===")
print(f"Perceptron (poly) CV: {acc_perc_poly:.3f}, Test: {test_acc_poly_perc:.3f}")
print(f"Pegasos (poly) CV: {acc_pega_poly:.3f}, Test: {test_acc_pega_poly:.3f}")
print(f"LogReg (poly) CV: {acc_logreg_poly:.3f}, Test: {test_acc_logreg_poly:.3f}")
=== Polynomial-Expanded Feature Set Accuracies === Perceptron (poly) CV: 0.926, Test: 0.924 Pegasos (poly) CV: 0.946, Test: 0.949 LogReg (poly) CV: 0.882, Test: 0.882
Compare to Linear (Non-Expanded) Results¶
We already have our linear results for Perceptron, Pegasos, and Logistic from previous cells:
accuracy_perc,accuracy_pegasos,accuracy_logreg(5-fold on linear features).
Let's create a small summary:
comparison_poly = pd.DataFrame({
"Linear": [accuracy_perc, accuracy_pegasos, accuracy_logreg],
"Polynomial(2)": [acc_perc_poly, acc_pega_poly, acc_logreg_poly]
}, index=["Perceptron", "Pegasos", "Logistic"])
print("Comparison of Linear vs Polynomial(2) feature sets:")
display(comparison_poly)
Comparison of Linear vs Polynomial(2) feature sets:
| Linear | Polynomial(2) | |
|---|---|---|
| Perceptron | 0.585118 | 0.925513 |
| Pegasos | 0.711005 | 0.945757 |
| Logistic | 0.717883 | 0.882041 |
Comparison Plot¶
models = ["Perceptron", "Pegasos", "Logistic"]
linear_accs = comparison_poly["Linear"].values
poly_accs = comparison_poly["Polynomial(2)"].values
x = np.arange(len(models))
width = 0.35
plt.figure(figsize=(8,4))
plt.bar(x - width/2, linear_accs, width, label='Linear')
plt.bar(x + width/2, poly_accs, width, label='Polynomial(2)')
plt.xticks(x, models)
plt.ylabel('CV Accuracy')
plt.title('Linear vs. Polynomial(2) Feature Expansion')
plt.ylim([0, 1])
plt.legend()
plt.tight_layout()
plt.show()
# comparison plot that shows the accuracy of the models on the test set
models = ["Perceptron", "Pegasos", "Logistic"]
linear_accs = [test_acc_perc, test_acc_pega, test_acc_lr]
poly_accs = [test_acc_poly_perc, test_acc_pega_poly, test_acc_logreg_poly]
x = np.arange(len(models))
plt.figure(figsize=(8,4))
plt.bar(x - width/2, linear_accs, width, label='Linear')
plt.bar(x + width/2, poly_accs, width, label='Polynomial(2)')
plt.xticks(x, models)
plt.ylabel('Test Accuracy')
plt.title('Linear vs. Polynomial(2) Feature Expansion')
plt.ylim([0, 1])
plt.legend()
plt.tight_layout()
plt.show()
Weights inspection¶
theta_perc_poly, theta_0_perc_poly = perceptron_train(
train_poly,
features = expanded_feature_cols,
target = target_col,
epochs = 5
)
print(f"Perceptron (poly) final model trained on entire train_poly. Bias (theta_0)= {theta_0_perc_poly:.3f}")
weights_df = pd.DataFrame({
"Feature": expanded_feature_cols,
"Weight": theta_perc_poly,
}).sort_values(by="Weight", ascending=False)
# Show top 10 positive weights and top 10 negative weights
print("\nTop 10 Positive Weights:")
display(weights_df.head(10))
print("\nTop 10 Negative Weights:")
display(weights_df.tail(10))
Perceptron (poly) final model trained on entire train_poly. Bias (theta_0)= 28.000 Top 10 Positive Weights:
| Feature | Weight | |
|---|---|---|
| 27 | x2*x9 | 108.472657 |
| 10 | x8 | 57.616740 |
| 20 | x1*x8 | 46.761335 |
| 12 | x9 | 24.724363 |
| 6 | x5 | 20.067918 |
| 0 | x1 | 17.537309 |
| 8 | x7 | 13.122208 |
| 44 | x3 | 11.087068 |
| 41 | x8*x9 | 7.061366 |
| 35 | x5*x8 | 6.677310 |
Top 10 Negative Weights:
| Feature | Weight | |
|---|---|---|
| 9 | x7^2 | -3.733855 |
| 19 | x1*x7 | -5.428044 |
| 43 | x9*x10 | -5.533118 |
| 5 | x4^2 | -6.143356 |
| 45 | x6 | -6.267363 |
| 13 | x9^2 | -6.325809 |
| 18 | x1*x5 | -7.307161 |
| 17 | x1*x4 | -12.237412 |
| 4 | x4 | -21.615008 |
| 31 | x4*x8 | -29.930359 |
theta_pegasos_poly, theta_0_pegasos_poly = pegasos_train(
train_poly,
features=expanded_feature_cols,
target=target_col,
lambda_param=0.01,
epochs=5
)
print(f"Pegasos (poly) final model trained on entire train_poly. Bias (theta_0) = {theta_0_pegasos_poly:.3f}")
weights_df = pd.DataFrame({
"Feature": expanded_feature_cols,
"Weight": theta_pegasos_poly
}).sort_values(by="Weight", ascending=False)
# Show top 10 positive weights and top 10 negative weights
print("\nTop 10 Positive Weights:")
display(weights_df.head(10))
print("\nTop 10 Negative Weights:")
display(weights_df.tail(10))
Pegasos (poly) final model trained on entire train_poly. Bias (theta_0) = 0.622 Top 10 Positive Weights:
| Feature | Weight | |
|---|---|---|
| 27 | x2*x9 | 2.558772 |
| 10 | x8 | 1.371412 |
| 20 | x1*x8 | 0.801088 |
| 6 | x5 | 0.613006 |
| 12 | x9 | 0.549907 |
| 8 | x7 | 0.470546 |
| 0 | x1 | 0.470503 |
| 16 | x1*x2 | 0.188492 |
| 3 | x2^2 | 0.186295 |
| 44 | x3 | 0.146303 |
Top 10 Negative Weights:
| Feature | Weight | |
|---|---|---|
| 30 | x4*x7 | -0.046538 |
| 24 | x2*x5 | -0.059779 |
| 43 | x9*x10 | -0.068885 |
| 1 | x1^2 | -0.079423 |
| 37 | x5*x10 | -0.086601 |
| 45 | x6 | -0.118576 |
| 5 | x4^2 | -0.124102 |
| 17 | x1*x4 | -0.215107 |
| 4 | x4 | -0.392167 |
| 31 | x4*x8 | -0.818844 |
Weights inspection: the three models compared¶
#Perceptron Weights
weights_perc_df = pd.DataFrame({
"Feature": expanded_feature_cols,
"Weight": theta_perc_poly
}).sort_values(by="Weight", ascending=False)
print("=== Perceptron (Polynomial) Weights ===")
print(f"Bias (theta_0) = {theta0_perc_poly:.3f}")
print("\nTop 10 Positive Weights:")
display(weights_perc_df.head(10))
print("\nTop 10 Negative Weights:")
display(weights_perc_df.tail(10))
#Pegasos Weights
weights_pega_df = pd.DataFrame({
"Feature": expanded_feature_cols,
"Weight": theta_pega_poly
}).sort_values(by="Weight", ascending=False)
print("=== Pegasos (Polynomial) Weights ===")
print(f"Bias (theta_0) = {theta0_pega_poly:.3f}")
print("\nTop 10 Positive Weights:")
display(weights_pega_df.head(10))
print("\nTop 10 Negative Weights:")
display(weights_pega_df.tail(10))
#LogReg Weights
weights_logreg_df = pd.DataFrame({
"Feature": expanded_feature_cols,
"Weight": theta_logreg_poly
}).sort_values(by="Weight", ascending=False)
print("=== Logistic Regression (Polynomial) Weights ===")
print(f"Bias (theta_0) = {theta0_logreg_poly:.3f}")
print("\nTop 10 Positive Weights:")
display(weights_logreg_df.head(10))
print("\nTop 10 Negative Weights:")
display(weights_logreg_df.tail(10))
=== Perceptron (Polynomial) Weights === Bias (theta_0) = 2.300 Top 10 Positive Weights:
| Feature | Weight | |
|---|---|---|
| 27 | x2*x9 | 108.472657 |
| 10 | x8 | 57.616740 |
| 20 | x1*x8 | 46.761335 |
| 12 | x9 | 24.724363 |
| 6 | x5 | 20.067918 |
| 0 | x1 | 17.537309 |
| 8 | x7 | 13.122208 |
| 44 | x3 | 11.087068 |
| 41 | x8*x9 | 7.061366 |
| 35 | x5*x8 | 6.677310 |
Top 10 Negative Weights:
| Feature | Weight | |
|---|---|---|
| 9 | x7^2 | -3.733855 |
| 19 | x1*x7 | -5.428044 |
| 43 | x9*x10 | -5.533118 |
| 5 | x4^2 | -6.143356 |
| 45 | x6 | -6.267363 |
| 13 | x9^2 | -6.325809 |
| 18 | x1*x5 | -7.307161 |
| 17 | x1*x4 | -12.237412 |
| 4 | x4 | -21.615008 |
| 31 | x4*x8 | -29.930359 |
=== Pegasos (Polynomial) Weights === Bias (theta_0) = 1.222 Top 10 Positive Weights:
| Feature | Weight | |
|---|---|---|
| 27 | x2*x9 | 5.118153 |
| 10 | x8 | 2.733301 |
| 20 | x1*x8 | 1.707763 |
| 12 | x9 | 1.111045 |
| 6 | x5 | 1.020537 |
| 8 | x7 | 0.792118 |
| 0 | x1 | 0.788783 |
| 44 | x3 | 0.319826 |
| 23 | x2*x4 | 0.226056 |
| 14 | x10 | 0.161368 |
Top 10 Negative Weights:
| Feature | Weight | |
|---|---|---|
| 36 | x5*x9 | -0.086738 |
| 42 | x8*x10 | -0.115658 |
| 37 | x5*x10 | -0.131599 |
| 19 | x1*x7 | -0.137374 |
| 1 | x1^2 | -0.158771 |
| 5 | x4^2 | -0.277386 |
| 17 | x1*x4 | -0.383653 |
| 45 | x6 | -0.488428 |
| 4 | x4 | -0.727732 |
| 31 | x4*x8 | -1.621416 |
=== Logistic Regression (Polynomial) Weights === Bias (theta_0) = 0.105 Top 10 Positive Weights:
| Feature | Weight | |
|---|---|---|
| 27 | x2*x9 | 0.885225 |
| 10 | x8 | 0.562952 |
| 20 | x1*x8 | 0.287563 |
| 8 | x7 | 0.275661 |
| 0 | x1 | 0.238675 |
| 6 | x5 | 0.232460 |
| 12 | x9 | 0.202625 |
| 3 | x2^2 | 0.195755 |
| 16 | x1*x2 | 0.169175 |
| 13 | x9^2 | 0.164768 |
Top 10 Negative Weights:
| Feature | Weight | |
|---|---|---|
| 32 | x4*x9 | -0.023050 |
| 33 | x4*x10 | -0.024077 |
| 36 | x5*x9 | -0.030435 |
| 37 | x5*x10 | -0.036984 |
| 24 | x2*x5 | -0.053120 |
| 18 | x1*x5 | -0.071317 |
| 14 | x10 | -0.081209 |
| 45 | x6 | -0.090833 |
| 4 | x4 | -0.227507 |
| 31 | x4*x8 | -0.245479 |
Plot¶
models = ["Perceptron (poly)", "Pegasos (poly)", "Logistic (poly)"]
test_accuracies_poly = [test_acc_poly_perc, test_acc_poly_perc, test_acc_logreg_poly]
plt.figure(figsize=(7,4))
plt.bar(models, test_accuracies_poly, color=["skyblue","salmon","palegreen"])
plt.ylim([0,1])
plt.title("Final Test Accuracies on Polynomial-Expanded Features")
plt.ylabel("Accuracy")
for i, v in enumerate(test_accuracies_poly):
plt.text(i, v+0.01, f"{v:.3f}", ha='center', fontweight='bold')
plt.tight_layout()
plt.show()
Kernel Methods¶
Kernelized Perceptron with Gaussian and Polynomial Kernel¶
k = 5
target_col = "y"
Hyperparameter tuning¶
sigma_list = [0.1, 0.5, 1.0]
epochs_kperc_gauss = [3, 5]
best_acc_kperc_gauss = 0
best_sigma_kperc = None
best_ep_kperc = None
for sg in sigma_list:
for ep in epochs_kperc_gauss:
acc = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=2, # small because even with k= 5 the result is the same. The result shown below is calculated with k=5.
random_state=42,
classifier_func=lambda tr, vl, f, t: kernelized_perceptron_classifier_func(
tr, vl, f, t,
kernel_func=gaussian_kernel,
kernel_params={"sigma": sg},
epochs=ep
)
)
if acc > best_acc_kperc_gauss:
best_acc_kperc_gauss = acc
best_sigma_kperc = sg
best_ep_kperc = ep
print("\nKernel Perceptron (Gauss) best => sigma=%.3f, epochs=%d, CV=%.3f" % (best_sigma_kperc, best_ep_kperc, best_acc_kperc_gauss))
best_hparams_table["KernelPerceptronGauss"]["sigma"] = best_sigma_kperc
best_hparams_table["KernelPerceptronGauss"]["epochs"] = best_ep_kperc
best_hparams_table["KernelPerceptronGauss"]["cv_accuracy"] = best_acc_kperc_gauss
Kernel Perceptron (Gauss) best => sigma=1.000, epochs=5, CV=0.915
Gaussian Kernel Perceptron¶
Traning set evaluation
print("=== Kernelized Perceptron (Gaussian Kernel) ===")
sigma_value = 1.0 # Adjust as needed
epochs_gauss = 5 # Adjust as needed
accuracy_gauss = k_fold_cross_validation(
df=train_scaled, # or use a smaller subset if runtime is high
features=final_features,
target=target_col,
k=k,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_perceptron_classifier_func(
tr_fold, vl_fold, f, t,
kernel_func=gaussian_kernel,
kernel_params={"sigma": sigma_value},
epochs=epochs_gauss
)
)
print(f"Gaussian Kernel: {k}-fold CV Accuracy = {accuracy_gauss:.3f}")
=== Kernelized Perceptron (Gaussian Kernel) === Gaussian Kernel: 5-fold CV Accuracy = 0.933
Polinomial Kernel Perceptron¶
Training Set Evaluation
print("\n=== Kernelized Perceptron (Polynomial Kernel) ===")
degree_value = 2 # e.g., quadratic
c_value = 1.0 # offset in (x^T y + c)^degree
epochs_poly = 5
accuracy_poly = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=k,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_perceptron_classifier_func(
tr_fold, vl_fold, f, t,
kernel_func=polynomial_kernel,
kernel_params={"degree": degree_value, "c": c_value},
epochs=epochs_poly
)
)
print(f"Polynomial Kernel (degree={degree_value}): {k}-fold CV Accuracy = {accuracy_poly:.3f}")
=== Kernelized Perceptron (Polynomial Kernel) === Polynomial Kernel (degree=2): 5-fold CV Accuracy = 0.925
Evaluation on Test set¶
Firstly, the Gaussian Kernel Perceptron
X_train = train_scaled[final_features].values
y_train = train_scaled[target_col].values
X_test = test_scaled[final_features].values
y_test = test_scaled[target_col].values
alpha_gauss = kernelized_perceptron_train(
X_train, y_train,
kernel_func=gaussian_kernel,
kernel_params={"sigma": sigma_value},
epochs=epochs_gauss
)
test_preds_gauss = kernelized_perceptron_predict(
X_train, y_train,
alpha_gauss,
X_test,
kernel_func=gaussian_kernel,
kernel_params={"sigma": sigma_value}
)
test_acc_gauss = np.mean(test_preds_gauss == y_test)
print(f"Gaussian Kernel Perceptron Test Accuracy: {test_acc_gauss:.3f}")
Gaussian Kernel Perceptron Test Accuracy: 0.938
Then the polynomial kernel
alpha_poly = kernelized_perceptron_train(
X_train, y_train,
kernel_func=polynomial_kernel,
kernel_params={"degree": degree_value, "c": c_value},
epochs=epochs_poly
)
test_preds_poly = kernelized_perceptron_predict(
X_train, y_train,
alpha_poly,
X_test,
kernel_func=polynomial_kernel,
kernel_params={"degree": degree_value, "c": c_value}
)
test_acc_poly = np.mean(test_preds_poly == y_test)
print(f"Polynomial (d={degree_value}) Kernel Perceptron Test Accuracy: {test_acc_poly:.3f}")
Polynomial (d=2) Kernel Perceptron Test Accuracy: 0.927
Kernelized Pegasos SVM¶
k = 5
target_col = "y"
Pegasos with Gaussian Kernel¶
lambda_val = 0.01
epochs_kpega = 5
sigma_val = 1.0
cv_acc_pegasos_gauss = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=k,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_pegasos_classifier_func(
tr_fold, vl_fold,
features=f, target=t,
kernel_func=gaussian_kernel,
kernel_params={"sigma": sigma_val},
lambda_param=lambda_val,
epochs=epochs_kpega
)
)
print(f"Kernelized Pegasos (Gaussian, sigma={sigma_val}) {k}-fold CV accuracy: {cv_acc_pegasos_gauss:.3f}")
Kernelized Pegasos (Gaussian, sigma=1.0) 5-fold CV accuracy: 0.856
Now on the test set
alpha_gauss, T_gauss = kernelized_pegasos_train(
X_train, y_train,
kernel_func=gaussian_kernel,
kernel_params={"sigma": sigma_val},
lambda_param=lambda_val,
epochs=epochs_kpega
)
test_preds_gauss = kernelized_pegasos_predict(
X_train, y_train,
alpha_gauss, T_gauss,
X_test,
kernel_func=gaussian_kernel,
kernel_params={"sigma": sigma_val},
lambda_param=lambda_val
)
test_acc_kernel_gauss_pega = (test_preds_gauss == y_test).mean()
print(f"Kernelized Pegasos (Gaussian) Test Accuracy: {test_acc_kernel_gauss_pega:.3f}")
Kernelized Pegasos (Gaussian) Test Accuracy: 0.846
Pegasos with Polynomial Kernel¶
degree_val = 2
c_val = 1.0
cv_acc_pegasos_poly = k_fold_cross_validation(
df=train_scaled,
features=final_features,
target=target_col,
k=k,
random_state=42,
classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_pegasos_classifier_func(
tr_fold, vl_fold,
features=f, target=t,
kernel_func=polynomial_kernel,
kernel_params={"degree": degree_val, "c": c_val},
lambda_param=lambda_val,
epochs=epochs_kpega
)
)
print(f"Kernelized Pegasos (Polynomial, deg={degree_val}, c={c_val}) {k}-fold CV accuracy: {cv_acc_pegasos_poly:.3f}")
Kernelized Pegasos (Polynomial, deg=2, c=1.0) 5-fold CV accuracy: 0.941
alpha_poly, T_poly = kernelized_pegasos_train(
X_train, y_train,
kernel_func=polynomial_kernel,
kernel_params={"degree": degree_val, "c": c_val},
lambda_param=lambda_val,
epochs=epochs_kpega
)
test_preds_poly = kernelized_pegasos_predict(
X_train, y_train,
alpha_poly, T_poly,
X_test,
kernel_func=polynomial_kernel,
kernel_params={"degree": degree_val, "c": c_val},
lambda_param=lambda_val
)
test_acc_kernel_poly_pega = (test_preds_poly == y_test).mean()
print(f"Kernelized Pegasos (Polynomial) Test Accuracy: {test_acc_kernel_poly_pega:.3f}")
Kernelized Pegasos (Polynomial) Test Accuracy: 0.945
Final Data Visualization¶
final_hparams_df = []
for method, params in best_hparams_table.items():
row = {"Method": method}
for k, v in params.items():
row[k] = v
final_hparams_df.append(row)
final_hparams_df = pd.DataFrame(final_hparams_df)
print("\n=== Final Hyperparameter Table ===")
display(final_hparams_df)
# (Optional) Save final_hparams_df if you want
# final_hparams_df.to_csv("best_hyperparams.csv", index=False)
=== Final Hyperparameter Table ===
| Method | epochs | eta | cv_accuracy | lambda | sigma | |
|---|---|---|---|---|---|---|
| 0 | Perceptron | 10.0 | 0.1 | 0.646638 | NaN | NaN |
| 1 | Pegasos | 20.0 | NaN | 0.724501 | 0.001 | NaN |
| 2 | Logistic | 10.0 | 1.0 | 0.717364 | 0.100 | NaN |
| 3 | PerceptronPoly | 10.0 | 0.1 | 0.925513 | NaN | NaN |
| 4 | PegasosPoly | 20.0 | NaN | 0.945757 | 0.001 | NaN |
| 5 | LogisticPoly | 10.0 | 1.0 | 0.882041 | 0.100 | NaN |
| 6 | KernelPerceptronGauss | 5.0 | NaN | 0.915131 | NaN | 1.0 |
| 7 | KernelPerceptronPoly | NaN | NaN | NaN | NaN | NaN |
| 8 | KernelPegasosGauss | NaN | NaN | NaN | NaN | NaN |
| 9 | KernelPegasosPoly | NaN | NaN | NaN | NaN | NaN |
# Example placeholders (DELETE or REPLACE with real computed values):
'''test_acc_perc_linear = 0.80
test_acc_poly_perc = 0.82
test_acc_perc_kernel_gauss = 0.84
test_acc_perc_kernel_poly = 0.83
test_acc_pegasos_linear = 0.81
test_acc_pegasos_poly = 0.83
test_acc_pegasos_kernel_gauss = 0.85
test_acc_pegasos_kernel_poly = 0.84
'''
# If you have logistic regression equivalents, define them too, e.g.:
'''test_acc_logreg_linear = 0.78
test_acc_logreg_poly = 0.80'''
# 1. We'll create a summary DataFrame
final_summary = []
# Perceptron
final_summary.append({
"Model": "Perceptron",
"Method": "Linear",
"Test Accuracy": test_acc_perc
})
final_summary.append({
"Model": "Perceptron",
"Method": "Polynomial Expansion",
"Test Accuracy": test_acc_poly_perc
})
final_summary.append({
"Model": "Perceptron",
"Method": "Kernel (Gaussian)",
"Test Accuracy": test_acc_gauss
})
final_summary.append({
"Model": "Perceptron",
"Method": "Kernel (Polynomial)",
"Test Accuracy": test_acc_poly
})
# Pegasos
final_summary.append({
"Model": "Pegasos",
"Method": "Linear",
"Test Accuracy": test_acc_pega
})
final_summary.append({
"Model": "Pegasos",
"Method": "Polynomial Expansion",
"Test Accuracy": test_acc_pega_poly
})
final_summary.append({
"Model": "Pegasos",
"Method": "Kernel (Gaussian)",
"Test Accuracy": test_acc_kernel_gauss_pega
})
final_summary.append({
"Model": "Pegasos",
"Method": "Kernel (Polynomial)",
"Test Accuracy": test_acc_kernel_poly_pega
})
# (Optional) Logistic
final_summary.append({
"Model": "Logistic",
"Method": "Linear",
"Test Accuracy": test_acc_lr
})
final_summary.append({
"Model": "Logistic",
"Method": "Polynomial Expansion",
"Test Accuracy": test_acc_logreg_poly
})
summary_df = pd.DataFrame(final_summary)
summary_df = summary_df.sort_values(by=["Model", "Method"]).reset_index(drop=True)
print("Final Test Accuracy Summary:")
display(summary_df)
# %%
# Barplot of final test accuracies by Model and Method
# Prepare data
models = summary_df["Model"].unique()
methods = summary_df["Method"].unique()
# Pivot the DataFrame for plotting
pivot_df = summary_df.pivot(index="Model", columns="Method", values="Test Accuracy")
display(pivot_df)
# Plot grouped bar chart
pivot_df.plot(kind="bar", figsize=(10,6))
plt.ylim([0, 1])
plt.title("Final Test Accuracies by Model and Method")
plt.ylabel("Accuracy")
plt.legend(title="Method", bbox_to_anchor=(1.0, 1.0))
plt.tight_layout()
plt.show()
# Single bar chart with all methods (Model x Method)
plt.figure(figsize=(8,6))
x_vals = range(len(summary_df))
acc_vals = summary_df["Test Accuracy"].values
labels = [f"{row.Model}\n{row.Method}" for idx, row in summary_df.iterrows()]
plt.bar(x_vals, acc_vals, color="skyblue")
for i, v in enumerate(acc_vals):
plt.text(i, v+0.01, f"{v:.3f}", ha='center', fontweight='bold')
plt.ylim([0,1])
plt.xticks(x_vals, labels, rotation=45, ha="right")
plt.title("Final Test Accuracies (All Experiments)")
plt.tight_layout()
plt.show()
Final Test Accuracy Summary:
| Model | Method | Test Accuracy | |
|---|---|---|---|
| 0 | Logistic | Linear | 0.733950 |
| 1 | Logistic | Polynomial Expansion | 0.882383 |
| 2 | Pegasos | Kernel (Gaussian) | 0.845917 |
| 3 | Pegasos | Kernel (Polynomial) | 0.945044 |
| 4 | Pegasos | Linear | 0.732922 |
| 5 | Pegasos | Polynomial Expansion | 0.949153 |
| 6 | Perceptron | Kernel (Gaussian) | 0.937853 |
| 7 | Perceptron | Kernel (Polynomial) | 0.927067 |
| 8 | Perceptron | Linear | 0.670262 |
| 9 | Perceptron | Polynomial Expansion | 0.923986 |
| Method | Kernel (Gaussian) | Kernel (Polynomial) | Linear | Polynomial Expansion |
|---|---|---|---|---|
| Model | ||||
| Logistic | NaN | NaN | 0.733950 | 0.882383 |
| Pegasos | 0.845917 | 0.945044 | 0.732922 | 0.949153 |
| Perceptron | 0.937853 | 0.927067 | 0.670262 | 0.923986 |
save